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ABSTRACT 

This study aimed at finding the opt imal condition of 
writteri iristructibn and online practice for learning a new computer 
applicatibn. Subjects were 44 undergraduate and graduate studen and 
university staff members. Those in the experimental group learned 
cbmmarids fbr an electrbriie spreadsheet by reading user-manual 
descriptibris arid wbrkirig training problems bhline. Training problems 
were varied iri order to_ cbritrbl how much independent problem solving 
subjects erigaged iri while learriirig ariy given command, and the forms 
bf practice iricluded pure guided practice, pure problem solving 
practice, arid mixed practice. Spacirig of the training problems was 
alsb man ipular-^.d . Subjects iri the control group read the same text 
arid studied the same trairiirig prbblems, but were hot permitted to 
type ariythirig at the keyboard during the training session. After a 
2-day delay, subjects iri bbth grbups solved new problems oh the 
computer withbut referericetb the iristruct ibrial materials. The 
results iridicate that prbblem sblvirig was a more difficult form of 
trairiirig thari guided practice, but it produced the best perfbrmahce 
at test. Distribut irig the spacirig bf trairiirig problems durihg 
trairiirig alsb _ improved perfbrmahce at test. These results have 
implicatibris for the desigri bf ihteractive tutorial manuals and for 
cognitive mbdels bf skill acq ^sitibri. Appended mater ials ihclude a 
typical mariual eritry fbr ari easy cbmmaridand a difficult commahd , a 
13-item refererice list, bne figure, and five tables. (DJR) 
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ABSTRACT 



This paper ainis at finding the optimal combination of verbal instruction arid 
on-line practice for learning a new computer application. Experimental 
subjects learned cbmmarids for ari electronic spreadsheet by. reading brief 
user-manual descriptions and working trairiirig prbblenis on-line . The form of 
the training problems was varied within subjects in order to control. how much 
independent, problem solving subjects engaged in while learning any giveri 
command.. Thero were three forms of practice: (i) Pure Guided Practice, in 
which subjects ere . told exactly what keystrokes to type to solve the 
problems; (2) Pure Problem Solving Practice, in which subjects solved 
problems without guidance; and (3) Mixed Practice, in which the first 
problem for a command was presented in Guided Practice form and two others in 
Problem Solving form. The spacing of the training problems was also., 
manipulated; th^^ problems . per taining to a given command were either Massed 
(i^e.; presented :orisecutively) , or Distributed (i.e., separated by other 
instructional raterial) . Aiiter a 2-day delay, subjects solved new problems 
on the computer without reference to the iristrUctibnal mater ials . _The 
results . indicate that problem solving was a more difficult form of training 
than guided practice^ but it produced the best performance at test. 
DistribUtirig the spacing of training problems during training also improved 
performance at test The results have clear. pragmatic implications for the 

^^_^^^^ctive tutorial manuals as well as implications for cognitive 
models of skill acquisition. 
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Abstr^^.ct 

This paper aims at finding the optimal combination of written Instruction arid bn-line 
practice for learning a new computer application. Experimental subjects learned commands 
for an etectrbnic spreacsheet by reading brief User-manual descriptions and working tra'ning 
problems on-line. The form of the training problems was varied within subjects in order to 
control how much Independent problem solving subjects engaged in while learning any given 
command. Th'jre vvere three forms of practice: (1) Pure Guided Practice, in which subjects 
were td)d exactly what keystrokes to type to solve the problems; ^2) Pure Problem Sdlving 
Practice, in which subjects solved problems without guidance; and (3) Mixed Practice, in 
which the first problem for a command was presented In Guided Practice form and two 
others in Problem S Mving form. The spacing of the training proble-ns was also manipulated; 
the problems pertaining to a given command were either Massed (i.e., presented 
consecutively), or Distributed (i.e.. separated by other instructional material): After a 2-day 
delay, subjects solved nev; problems on the computer without reference ?o the Instructional 
materials. The results inc'icate that problem solving was a more difficult "form of training than 
guided practice, but it produced the best performance at test. Distributing the spacing of 
training problems daring training also improved perfdrmance at test. The results have clear 
pragmatic implications for the design of interactive tutorial mianUals as well as implications 
for cognitive models of skill acqulsitidh. 
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Charriey - Interactive Tutorials 2 

Designers of Instructional manuals fc computer user's are beginning to recognize the 
importance of cognitive models of text comprehension snd skill learning for making decisions 
about such issU3s as informatidn content, organization, iaydut, and so on. This trend is all 
to the good: however, [Droblerhs arise \/hen different learning paradigms lead tc conflicting 
pririclpies for text design. For exarnple, two powerful ieirnirig strategies that ha^ e received 
great attention m recent years are learning from examples and discoveiy iearni ig. These 
strategies suggest quite different approaches to the conteni of Instructionai tex:s. The 
learning from examples approach suggests that the text provide numerous worked-oui 
examples that learners can use as models for snlving problems on their own. Sweller and 
Cooper (1985) advocate this approach for Teaching math stUderKS to apply algebra 
procedures appropriately. In contrast, Carroll. Mack, Lewis, Grischkowsky dnd Robertson 
(1985) argue that people learning to us3 a word processor should set their own goals and 
explore the workings of the systemi with minimal guidance from the text, Carroll and his 
colleagaes even suggest omitting inforriation about some features of the system from the 
manual, in order to force learners to discover the information Independently. 

Carroii et al:'s (1985) discovery learning approach is a direct response to two aspects of 
standard commercial manua! design. The first aspect concerns the degree of explicit 
explanaiidn. In the traditional view, instructional manuals for novice users should be as 
complete and explicit as possible, cdntaihing detailed explanations of every relevant concept 
anu procedure (Tausworthe, 1979; Price, 1984). However, as many people have observed, 
computer users generally dislike reading long, detailed manuals: they prefer to have someone 
show them what to do or figure things out for themselves (Draper, 1984; Scharer, 1983; 
Carroll, 1984): Carroll (1984) found that learners actually performed better on on-line tests 
after studying drastically shortened manuals that eliminated laige sections of explanation and 
elaborations than after studying a fuli-length commercial manual. In contrast, Reder, 
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Charniay Interactive Tutorials 3 

Charney. and Mdrgan (1986) prepared two versions of a riianuai for the PC-DOS operating 
system, one version vith elaborations and the other without. Subjects who studied the 
elaborated version pierfbrmed better on an on-line test than subjects who studied the 
unelaborated manual. However, when the availability of elaborations vvas ccntrdlied by type, 
experienced and Inexperienced computer users were found to benefit only from selected 
types of elaborations. We will return to these results below. 

A second common feature of commercial manuals is the so-called "tutorial" section that 
provides users with directed, step-by-step, hands-on practice. In many respects, tutorials are 
quite similar to the presentation of wdrRed-but examiples; the major difference Is the addition 
of the motor activity of carrying but the step-by-step instructions and viewing the system's 
prompts arid feedback. Carrcll et al. (1985) argue that people learn skills best through 
exproratioh arid that instead of describing each system feature expilicltly and guiding learners 
through tutorial exercises, mianuals should leave many details of the system to be discovered 
by the learner. In a study of people learning to use a text editor, Carroll et al, 
demonstrate the sup3riority of a "Guided Expldratidn" manual over a tutorial manual. The 
Guided Exploration materials described each prdcediire very briefly, with minimal explanation, 
and even omitted certain prdcedUral details. Subjects using the Guided Exploration materials 
set their own "prdblems" (e.g., deciding to compose and print a letter), and executed 
p dcedures at their own initiative. In contrast to people working through a commercial tutorial 
manual, subjects who worked with the Guided Exploration materials spent less time dri 
training, less time on the test (typing and printing a letter), and Used the procedures more 
successfully (fewer failed attempts at executing a cdmmand). 

We share the intuition that people who follow Instructions step-by-step dften work 
mechanically, without thinking enough about the purpose df each action. However, it is not 
clear that discovery learning Is the best alternative td step-by-step instruction or indeed thai 
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it was the^ primary cause of the Irriprbvemcht that Carroll et al. (1985) fburid. As the 
authors themselves point but. there were substantial differences in the infbrrnation conisnt, 
clarity, and organization of the two manuals, so it is difflcali to know how much the 
discovery aspect of the experimental materials contributed to the results. Second, discovery 
learning was confounded In this study with problem solving activity. The subjects In the 
exploration group not on'y set their own goals, they also worked independently to solve 
them. Since problem solving is itself a powerful learning tool, it may have been this rather 
than the goal setting and exploration that contributed most to learning. 

There are several reasons why discovery learning may not be the optimal way tc le^arn a 
hew system. An irnportant part of learning a new system is knowing when any given 
procedure is most appropriate (Charney & Reder, in press; Reder, Charney & Morgan, 
1986). Thic Rind of knowledge is often difficult to acquire without previous exposure to the 
problem situations that may com.'ionly arise: People learning an operating system or a 
programming language might have difflcul; ' setting reasonable goals that really explore the 
system's capabilities. For example, a person who views a text editor as a glorified 
typewriter may not be able to Invent a problem involving multiple windows or search-and- 
replace fUnctidns. Further, inexperienced users may never invent a problem for themselves 
that demonstrates the advantages of one procedure over another. As we argue elsewhere 
(Charney & Reder, in press), unless learners are presented with the situations that motivate 
the use of one procedure over another, they are likely to stick with some procedure the> 
find most memorable, regardless of Its efficiency. Finally, ncvice uners who learn drlrharily 
through explor'ng a system may develop and retain serious mlscbhceptibns Unless their 
exploratioh leads to a highly salient error or prbblernatic result (Neuwirth. 1985). 

Sweller and Cooper (1985) have similar doubts about problem solving itself, v;hich Is an 
integral piart of discovery learning. Sweller and Cooper argue that learners benefit more from 
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studying worked-dut examples than frdrh prbbiern solving, at least early in the teaming 
process. Problem solving, they contend, interferes: with the acquisition of schemas of 
prbblerri types which are necessary ror learning to use procedures appropriaiely: For 
example, an expert at solving algebra problems has schernas for discriminating between 
problem types and knows equations that are suitable for each type: Unless novices acquire 
similar schemas. tiiey must fail back on standard problem solving search techniques to find 
appropriate procedures to use. They may spend a long time on fruitless solutibh paths 
without ever discovering the "right" one and may never extract a general rule from their 
experience. Studying wbrRed-out examples of math problems gives students the necessary 
ihfbrrhatibn for discriminating types of problems and allows them to build appropriate 
schemata. Consistent with this analysis, Sweller and Gooper found that math students who 
studied four example problems and solved four problems during training were faster and 
more accurate at solving new problems at t3St than students who trained by solving eight 
problems. 

The apparent conflict in the results from the discovery learning and learning by example 
paradigms may be due to a difference In what aspect of skill learning is Under investigation. 
Studying examples may be important for certain cdmpdhents of skill learning, while discovery 
learning may be impdrtanl for others. To explore this possibility, we must consider more 
carefully the types of learning that go on In initial skill acquisition. We conceive of initial 
skin learninQ as consisting of three critical components (Reder, Charney & Morgan, 1986; 
Charney & Reder. in press): 

o learning hovel concepts and the functioHaiity of ncvel procedures; 

• learning how to execute the procedures; 

• learning the conditions under which a procedure Is applied; and remembering the 
best procedure to execute in a given situation. 
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Carroll et al. (1985) seem to be concerned primarily with the first two compdriehts: 
discovering what options/featares are available on a system and how to implement them. 
PerhafDS becaase the execution of mathematical dperatidns is a well-learned skill for high 
school stadents. Sweller and Cooper (1935) seem mdre cdncerned with the third component, 
recognizing which procedures are appropriate In particular situations. In our earlier research 
(Reder, Charney & Mdrgan, 1986). we found that learners benefited from elaborations and 
examples concerning the second component, namely how to execute procedures, but not 
from elaborations on the concepts or when to apply procedures. Our results suggested that 
manuals with examples of correct commands are superidr td manuals without such 
elaborations, but we had not contrasted reading examples against carrying out tutorial 
exercises or against problem solving (with or without a discovery component). 

The present experiment contrasted various cdmbinatidns of written instruction and active 
problem solving during training in learning n cognitive skill, namely, learning to use an 
electronic spreadsheet. To learn some spreadsheet commands, subjects worked through 
solutions td training problems step-by-step in a tutorial format (Pure Guided Practice): For 
other commands, they solved training problems independently (Pure Problem Solving) For a 
final group of commands, subjects received a combination of Guided Practice and Problem 
Solving (Mixed Practice): Finally, a group df Cdntrdl subjects studied the complete sei of 
training probler^s, all with explicit solutions, but did not type anything at the computer. 

We expected that Guided Practice would be of greater benefit to novice computer users 
than simply studying example prdblems and their sdlutions (Control group). " Novices often 
have difficulty understanding the purpose of a procedure, or the significance of different 
parameter specifications until they have seen the procedure demoristrated. Studying the 
steps of an example problem without actually carrying them out may not be sufficient fdr 
comprehension and long term retention: As noted above, carrying out step-by-step 
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instructions also provides information about tne system's prompts and ("eodback. Wo 
expected Guided Practice to exhibit some of the benofit^-: of exampios; sucli as helpinc] 
subjects I'jam to distinguish between proceduips. On fhe other fiancl, we expected Problem 
Solving practice id help subjects learn how to cjencr^ite correct sequences of operations, 
which they may not learn well unless thov are facod with problems to solve without 
guidance. Since guidance seems important for some aspects of skill learning, and 
independent problem solving seems Important for otlK^rs, the best perfcrmnnco might arise in 
the Mixed Pract'ce condition, ./hicli provides an initial problem in tutorial form fc.lowed b\ 
opportunities to solve novel problems IndependL i<ly. 

Before describing the study in detail, we would like to briefly highlight the ways in which 
it differs from Sweller and Cooper (1985) and Carroll et al. (1985) The design of our 'y 
differs frdrh that of Sweller ard Cbbper in several important respects. First. Sweller and 
Cooper did not have a condition analogous to bur Gui'derl Practice ccnditibh. Their subjec;s 
studlec Example algebra problems but did not work through them with step-by-sfep guidance. 
Second, Sweller and Cooper did not have a pure example training conaition. in order to 
moiivate their subjects to attend to the examples, they paired each workeo-out example with 
a similar problem to solve. Finally. Sweller and Cooper's subjects were tested ifnmediately 
after the training session, while we imposed a two-day delay between training and testing. 
Our study rinakes a more rigorous cbrhparisbh of learning by exarhples and probieni sblviiVg 
because it includes both pure and mixed training conditions. We imposed a delay between 
training and test because we suspected that the benefit of problem solving may increase 
with time, as retrieval of particular exan^ples from memory becomes more difftcuit: 

Like Carroll et al. (1985), our sti'dy con^^'asts tutorial practi::c against problem solving for 
learning a com.puter applicaticn: However, our design differs from theirs in two major ways. 
First, we h^-'d the Information content of the manuals constant across conditions and only 
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varied the type of practice subjects engaged in. Second, our subjects were provided with 
opportanities for problem solving but not for discovery learning; Oar claim is that a 
sabstantial part of the advantage that Carroll et al. attribute to discovery learning may be 
due simply to the prbblerTi solving activity their subjects engaged in. We do hot test this 
claim directly since we do not have a discovery learning training condition However, our 
claim will be indirectly supported to the extent that problem solving without goal setting and 
exploration produces better performance than pure guided practice. 

Our study also had a secondary goal that neither of the other studies shc-.red, namely, to 
extend to the cognitive skill domain the well-known result of superior learning with distributed 
practice; In numerous verbal learning experiments, subjects have been found to perform 
better on recall tests if the training list distributed the repetitions of the items across the list 
than if the repetitions appeared in the list consecutively (see Glenberg, 1979 and Hintzman, 
1976 for reviews). We orthogonally varied the spacing of practice and the type of practice 
to see whether increased problem-solving activity attentuates the benefit of distributed 
practice. 

Method 

Design 

There were two groups of subjects, one considered experimental and the other control: 
The experimental subjects learned commands for the VislCalc electronic spreadsheet by 
reading manual entries and working practice problems on the computer. The control 
subjects read the same text as the experimental group and studied the same training 
problems ^all with explicit solutions), but were not permitted to type anything at the keyboard 
during the training session; 

For all subjects, the text describing the commands was held constant, but the nature of 



13 



eharney - jnteractive TDiorials S 

the training problems and their spacing varied orthbQbrialiy. The Spacing factor determined 
whether the training problems for a given cummarid were presented consecutively (Massed), 
or whether text and proolems concerning other commands intervened (Distributed), The 
Spacing variable was a within subject variable, but the Practice Form variable was partly 
b(5tweeh and partly within subject. The control subjects saw only one form of training 
problem, that is. with an explicit wbrked-but solution. The experimental subjects saw 
different forms of training problems for diffe. .rit commands. There were three forms of 
practice: (i) Pure Guided Practice, in which subjects were told exactly what ReystrdRes to 
type to solve three training problems; (2) Pure Problem Solving Practice, In which subjects 
solved the training problems withoot goidance; and (3) Mixed Practice, in which ihe first 
training problem for a command was in Guided Practice form and the remaining two 
proL ems were in Problem Solving form. 

The lest factor, Command Difficulty, was within sobject, varying for both the experimental 
and control groups. The command? were classified into two groups, "difficult" and "easy:" 
Cbmmarids of both difficulty levels were randomly assigned to Practice Form and Spacing 
conditions for each subject. 

For both experimental and control groups, learning was measured in a delayed test in 
which subjects solved additional problems on the computer without reference to the training 
materials. The main dependent measures were success at solving the problems arid time at 
task. 

PJI ate rials 

Individaal training manuals were prepared for each subject. Each manual began with a 
general introduction to the VisiSalc electronic spreadsheet (725 words), including Instructidri 
in scrolling the spreadsheet and moving the cursor: Next came brief descriptions (averaging 
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250 words) of 12 VisiCalc cbrrinriands (e.g.. entering and formatting data, deleting rows, 
replicating entries). Six ebmmands were classified as difficult because their syntax was 
cornplex: and six. with relatively simple syntax, were ciassified as easy. These are listed iri 
Tsble 1. The 12 commands were independent in the sense that learning the syntax ^br brie 
would not facilitate acqulsltidn of another. The descriptions contained explicit information 
about the pL'pdse of each command and an abstract rule for the command syntax; The 
parts bf the rule were explained in detail, but no examples r. correct commands we^e 
provided. A typical manual entry for an easy cbmmand (Blank) and a difficult cdmmarid 
(Move) appear in Appendix A. The manual entHes for each command were followed by brie 
or more training problems, presented individually on separate pages. 

INSERT TABLE 1 ABOUT HERE 



rour problems were constructed for each command; one was randbmly selected to be 
the final test item for a given subject and the remaining three appeared in the manual as 
training problems. A typical problem presented the subject with a previously prepared 
VisiCalc spreadsheet on the screen of the computer and a goal for how the spreadsheet 
should be modified. The sdlutidn vd the problem required just one of the 12 target 
commands, but may have also required standard scrdlling and cursdr mdvement dperatlons. 

The experimental design required two versidns df the training problems: a version that 
[Dresented a step-by-stejD sblutibn (Guided Practice form) and a version that simply presented 
a goal that subjects were to achieve on their own (Prbblem Solving form) J Figure 1 
presents the two versions of the instructions for a problem pertaining tb the Move comrriarid 
and the associated VisiCalc display. 
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INSERT FIGURE 1 ABOUT HERE 



As indicated in the part A of Figure 1, the Guided Practice fbrm of a problem stated 
tht; goal without naming a specific command for achieving It. The goal was followed by the 
Ino'rruction "TYPE THIS" and a solution to the problem, presented as a sequence of keys to 
press. In Problem Solving form (part B of Figure 1), the wording of the goal was preserved 
as far as possible, and the instructions for what to type were simply ornltted. All training 
problems that required problem solving prov.ued feedback on the next page. In order to 
keep constant the amount and kinds of presented informiation across conditions, the 
feedback consisted of the sequence of keystrokes provided in the Guided Practice version of 
the problem. - To avoid mentidning the names of the commands directly, the instructions 
occasionally included a diagram of the goal-state of the spreadsheet: In the Problem Solving 
form of these problems, the diagrams appeared on the same page as the instructions, but 
in Guided Practice formi: they appeared as feedbaci; on the following page. 

The version of a training problem that a subject saw varied with condition. Subjects 
assigned to the control group always saw training oroblems in Guided Practice fdrrh, but 
they did not carry out the soiut'ons, they merely smdied thertl. For each subject in the 
experimental group, commands were randdrhly assigned to Practice Forms, with the 
constraint that four commands were assigned to Pure Guided Practice, four to Pure Probjem 
Solving Practice and four to Mixed Practice. The assignment was further constrained to 
ensure that two of the commands assigned to os»-h form were difficult and two were easy: 

As described In the Design paragraph, Spacing of problems was a within subject 
variable. For cdrhrhands assigned to the massed condition, the three training problems 
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apfDeared immediately after the reievarit manuai entry. For cbrnmands In the distrl-^'i^^d 
condition, one problem appeared imrnediaiely after ihe relevant manual entry, and the 
remaining two problems appeared later in the manual after entries and problem:? pertaining 
to other commands. The amount of Intervening material between the distributed prdblems 
ranged from 1 new manua! entry and 3 non-pertinent prdblems to 3 manual entries and 7 
prdblems. Td ensure fdr adequate spacing of the cbmmands, we constructed a 
massed/distributed spacing template. Although the presentation order of commands was 
randomized for each subject, the template determined the relative positions of training 
problems within the manual: 

In addition to the training manual for each subject, a test booklet was constructed that 
contained the 12 problems that remained after the practice problems were chosen fcr each 
command: The test problems were randomly ordered and each problem appeared on a 
separate page of the test booklet. The Problem Sdlving fdrm df the prdblens were Used, 
except that nd feedback \A/as prdvided. 

Subjects 

Forty-four members of the Carnegie-Melldn University cdmrriunity (undergradUales, staff 
and graduates) participated in the experiment. Subjects varied i i prevlbUs computer 
experience from novice to experienced computer user. A questionnaire was used to rate 
subjects' experience and to screen out subjects who were familiar with electronic 
spreadsheei programs: (This will be discussed further in the results section.) Subjects were 
paid at a rate of $3 per hour for participating in the experiment or received a combination 
of money and course credit for a psychdiogy class. 

Procedure 

Subjects were run Individually in twd sessidns: a training session arid a testlrig sessidri 
twd days later. 
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At the start of the trair g sessibri; sobjects were seated before an IBM Personal 
ebmputer (iBM-F^e) displaying a blank VisiCalc spreadsheet. The training manual rested bh 
a iectarn beside the IBM-PC. Subjects were instructed to read the rhanual one page at a 
time, without turning back to previous pages. Subjects in the experimental group were told 
that the manual would include training problems that they would solve on the cbrrifDUter and 
that they were perrnitted to type at the keyboard only while working on a problem. When 
the instructions for a training problem included the words "TYPE THIS," they were to type 
in the exact sequence of keys Indicated: When the instructions did not provide a solution, 
they were to sblve the problem on their own as efficiently as possible. Subjects in the 
cbntrbi group were told that the manual contained descriptions of VisiCalc commands 
followed by problems exemplifying how to issue the cdmrhands. They were to study the 
examples, but not type anything at the keyboard. Subjects worked through the manual at 
their own pace] on the average, the experimental subjects took 1-1/2 hours and the control 
subjects, 1 hour. 

Two black-and-white video cameras were used to record the subjects' interactions with 
the computer and to collect reading and problem sblving times. One camera was focused 
on the manual and one on the screen of the IBM-PC: A mixer connected to a video 
cassette recorder produced a split image allowing us to record the top few lines of each 
manual page concurrently with the VisiCalc display on the screen. A rhillisecdnd timer was 
superimposed In the lowe'* right corner. This allowed us to record and time the subject's 
interactions with the computer, as well as reading times per page of the manual. 

Two days after the training session, subjects returned for the testing session in which 
they solved one problem for each command without feedback and without reference to the 
instructidhal text. The overall procedure for the testing session was Identical to the training 
session; most subjects completed the test in less than 4G minutes: The experimenter was 
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present durrng bd»h sessions to call up the appropriate VIsiCalc spreadsheet for each 
problerh and to note the subject's success at working the problems. 

Results and Discussion 

Scoring 

We used several performance measures. One was success at solving the prbblehis 
(accuracy). We awarded 1 point for each correct solution ^nd 0 points for an incorrect 
solution: The correctness of a solution was judged by whether it satisfied the goals 
specified for the problem, using appropriate commands; Partial credit {;5 points) was 
awarded If the subject attempted to use the appropriate command but missed rome minor 
detail of the syntax. We also noted sdiutidn times for correct and partially correct 
responses. These were calculated fron the timestamps bri the frames of the videotape, 
measured from the first appearance of the page displaying the instructions for the problem 
to the time when the last command of the solution was entered. Although the data in 
Tables 2 and 5 report soiotion times in seconds, the analyses used /og{time) In order to 
normalize the data. 

Subjects' previous computer experience was rated on the basis of a questionnaire: 
Subjects were asked to list the types of ccnjputers they had used, to list the programming 
rahguages they had studied and the duratldn of study, and to rate the frequency with which 
they used various computer applications (e.g., text editors, graphics packages, electrdnic 
spreadsheets, and statistical packages). Subjects who were familiar with electronic 
spreadsheets did not participate in the study. The remaining subjects vvere classified into 
three categories. They were rated as experienced computer users if they were famliiar with 
two or more computer operating systems, hod studied two or more programming languages 
for a total of at least one year, and were irequent users of text editors. intermBdiate 
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computer users were familiar with one or two computer systems, had one or two semesters 
of computer programming and had used computers less frequently in general; as well as 
less frequently for text editing. Subjects with even less experience or no experience with 
computers were rated as novrces- 

The mean experience level of the control and experimental groups was computed using 
a three-point rating scale: 0 (novice), 1 (intermediate), and 2 (experienced). Subjects in the 
experimental group had a mean rating of 1.0, and control subjects had a mean rating of 
1.3. Experience was used as a grouping variable in all of the analyses to be reported. The 
results validated the ratings: subjects wli.i jr ;ater experience solved more problems correctly 
and tended to work more quickly. However, computer experience did not interact with any 
variables of interest: So, to simplify the exposition, the data to be reported are collapsed 
over experience levels. 

The results also validated the categorization of the commands by difficulty: difficult 
commands had significantly longer solution times at test than easy commands: However, 
Command Difficulty, like Computer Experience, did not interact with any other factors of 
interest. So we have also dropped the Difficulty factor from the discussion of this study. 

The analyses for the experimental group are therefore the result of 3x2 ANOVAs over the 
Practice Form and Spacirr factors. To conpare the control and experimental grdups, we 
collapsed the data from the experimenta' group over Practice Forrtis and fDerfbrmed a 2x2 
ANOVA, with factors Spacing (Massed vs. Distributed) and Group (Experimental vs. Control). 

Performance on Problems During Training 

Table 2 presents mean accuracy .scores for the experimental group bri the training 
"oblems as a function or Practice Form and Spacing. Only data for the experimental 
lup are reported for this measure, since control subjects merely studied the problems 
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without interacting with the computer. There was a significant main effect of Practice Fbrrti 
on accuracy, F(1.27)= 19.9. p<.01, with better perfdrmahce in the Guided Practice cbhditibri 
than in the Mixed and Problem Solving conditions. Accuracy scores on Guided Practice 
problems were about 15 percentage points higher than on Mixed Practice and Problem 
Solving problems; the latter two conditions did not differ significantly; 

The Spacing of the training problems in the training manual also Influenced perforrriance: 
Overall, subjects were 5% more accurate when problems were massed than when they were 
distributed, and this difference, thoagh small, was reliable, F(1,27) = 4;3, p<.05. 

INSERT TABLE 2 ABOUT HERE 



Similar results were obtained for solution timos. Table 3 presents the mean tiroes (in 
seconds) that subjects took to correctly solve a training problem: Practice Form again 
produced a significant main effect. F(2,54)= 17:9, p<:ei: Subjects solved Guided Practice 
problems about 20 seconds faster than Mixed Practice problems, ?(29) = 3.7, p<.01, and 
about 30 seconds faster than Problem Solving problems, ?(29) = 5.0, p<.01. Subjects were 
also faster at .Solving the Mixed Practice problems than the Problem Solving problems, 
?{29) = 2.9. p<.01. 

There was no main effect of SfDacing on sblutidn times. Subjects appeared to take 
longer oh massed problems than distributed problems in the Problem Solving condition; 
however, the interaction of Spacing arid Practice Form was not reliable.^ 

Table 3 also lists the mean time that control subjects chose to spend studylhg e training 
problem and its sblutidn. These times do not represent Interactions with the computer, 
since subjects in this condition were not allowed to type at the keyboard: Rather, these 
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data represent the rheah amnurit of time that subjects studied the page of the manual 
cbhtaihihg a training problem. It is interesting to note that the times for the control group 
are quite similar to the times in the Guided Practice condition. The similarity of the Guided 
Practice condition to the control group will be discussed further below. 

INSERT TABLE 3 ABOUT HERE 



We are not inclhied to give too much weight to the finding that Guided Practice 
produced the best results during training; in order to produce a correct solution to a Guided 
Praciice problem, subjects simply had to foliow the instructions. Carroll et ai. (1985) 
observed that step-by-step instructions sometimes caused difficulties when subjects made 
mistakes or explored on their own. We aid not observe subjects having difficulty following 
the Guided Practice instructions or getting back on track If they made an error. However, 
the presence of the experimenter may have led subjects to follow the Instructions more 
carefully and discouraged them from exploring on their b/vn. 

The superior perfbrrriance on massed proLjIems during training is also to be expected. 
In the massed condition, the solution to a problem can be held in working memory and can 
serve as a model for subsequent similar problems. Therefore, it is of greaier interest 
whether corriparable results are obtained at test. 

Performance on Problems at test 

Table 4 presents the mean accuracy scores for test problems, as a function of Practice 
Forrn and Spacing: Although Practice Form again produced a significant main effect, 
F(2,54) = 4.8, p<.05, the source of the effect is quite different: The Guided Practice 
condition produced the worst performance at test rather than the best. Since accuracy was 
very similar In the Mixed and Problem Solving conditions, the effect of Practice Fbrhi Is 
clearly due to the superiority of these two conditions over the Guided Practice cbhditfbh. 
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Unlike performance daring training, accuracy at test was hot Irifluenced by the Spacing 
of tl'ie irairiing problems, f'.-=2:0: 

INSERT TABLE 4 ABOUT HERE 



Table 5 presents the mean solution times for correct responses. There was no main 
effect of practice Form on solution times; subjects in all three conditions solved problems in 
about 90 sec. Taken together with the accuracy data, these results suggest that the lower 
accuracy of the subjects in the Guided Practice condition was clue to a real difference in 
learning and not simply to a speed/accuracy trade-off. 

There was a main effect of Spacing bri solution times, F(1.27) = 9.5, p<;01; As shown in 
Table 5, distributed spacing shortened overall solution times by an average of 24 seconds. 
We had expected that increased problem solving activity in the Problem Solving and Mixed 
Practice conditions would reduce the benefit of dijJtributed spacing as compared to the 
Guided Practir-e condition: Contrary to our expectat^dns. though, the spacing effect appears 
to be largest in the Problem Solving arid Mixed Practice conditions. However, the 
inieraction of Practice Form and Spacing was not statistically reliable, F=1.9. 

INSERT TABLE 5 ABOUT HERE 



The Control Group vs. the Experimental Group 

dverall, performance of subjects /i the experi- .ital group was superlnr to that of the 
control group, those subjects who studied the training nianual without interacting at all with 
the computer. Their data are given in the rightmost column of Tables 4 and 5. By 
collapsing over Practice Form, a 2x2 ANOVA was performed, using as factors of treatment 
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Spacing (Massisfd vs. blstrlbutedj and Group (Experimental vs. Controi). The experimental 
group solved more test problems correctly, F{1,38) = 5.C!, p<.01, and was faster at solving 
the problems than the cbritrol group, F(1,36) = 8.6, p<.01.^ There was no main effect of 
Spacing on either measure. One might have expected an interaction between Spacing and 
Group, since Spacing had produced a main effect on sbiutidn times for the experimental 
subjects: However, the Spacing x Group interaction was not reliable for either solution times 
(F(1,36) = 2.4, p>.1) or accuracy (F.-1). 

Although the performance of the experimental group was superior to the controi group 
overall, the performance of the experimental subjects in the Guided Practice condition was 
very similar to that of the control group. As noted above, subjects in the two cdnditidns 
spent equal ambUrits of time studying the training prbblems. The two cbriditibhs also 
produced similar levels of accuracy at test. However, there appears to be an advantage for 
Guided Practice training over simply reading, in terms of solution times at test: the control 
group spent 135 sec: per problem, while the mean solution time in the Guided Practice 
condition was only 96 sec: The faster time in the Guided Practice condition may be due to 
the hands-on, step-by-step interaction with the computer that this condition provided. 
However, this is not the only possible explanation. Subjects in the Guided Practice 
condition were part of a withiri-sUbject design and thUs were also exposed to problem-solving 
training for other cbrYirnahds. This problem solving practice may have contributed to faster 
times overall. To resolve this point, we would have to use a betv/een-subjects design, to 
see wheiher subjects trained Exclusively with Guided Practice woula still produce faster times 
than the control group; 

The Disadvantages of Guided Practice Training 

Why was Guided Practice training less effective than training that Included Problem 
Solving? One possible explanation involves the amount of time subjects spent on ihe 
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training problems. Subjects spent the least amount of training time on prdblerhs in the 
Guided Practice cdnditidn and the most tirhe on Problerh Solving problerhs. One could 
argue that the forrri of practice Is irrelevant excefDt insofar as it motivates subjects to spend 
more time bri training, thereby p>rbducini3 a stronger lepresentation of the procedures in 
memory. 

Two features of the results are inconsistent with this interpretation. First, even when the 
time spent on training was equivalent, the type of training stiN influenced performance; In 
particaj?.r, the control group put in slightly more study time per problem than subjects in the 
Guided Practice condition, but the Guided Practice cdnditidn prdduced significantly shdrter 
solution times at test. Second, differences in study time did not always produce 
cdrrespdriding differences in perfbrmance. Subjects spent significantly /ess time during 
training on Mixed Practice jDroblems than Problem Solving problems. Rowever, at test, the 
two conditions did not differ significantly in terms of eithei accuracy or solution times: Of 
coarse, the shorter training times for trie Mixed condition were probably due to the fact that 
the first mixed trial was guided practice, which subjects completed quickly since it did not 
require probierti solving. 

In order to further refute the argument that our results are confounded with total training 
tim3, we computed correlations between training time and test perfdrmance fdr each df the 
three types df training. The cdrrelatidns between iralning time and test accuracy tended to 
be small and negative {Guided Practice cdnditidn, r= -.02; Problem Solving condltidn, r= 
-.29; Mixed Practice condition, r= -.12), suggesting, if anything, that subjects who spent 
more time on training problems tended to perform less accurately at test. The correlations 
between training time and solution times at test tended to be small and positive (Guided 
Practice, r= .22; Problem Solving, r= .21; Mixed Practice, r= .13), which may simply reflect 
dif'arences in typing speed among subjects. 
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• Thus, while the evidence is indirect; ihere seems to be: safficient reason to Believe thai 
the difference between the Saided Practice condition and the conditions that ihciude 
Probjem Solving are due to the quality of the activities thc^t subjects perform durng tralnlhg 
and not simply ?he quantity of time and attention they devote to training. 

Problem Solving vs. Mixed Practice 

is Mixed Practice just a mixture? We driglnally included this condition because we 
suspected thai subjects would need an example of a correct command in order to interpret 
the abstract syntactic rUle (Charney S Reder, In press; Reder, Charney & Morgan, 1986); 
We expected subjects to benefit from seeing one Guided Practice proBlern before trying 
[Droblems on their own. Rowever, the results reveal very little difference between the 
Problem Solving and Mixed conditions: During training, solution times were snorter in the 
Mixed condition; however, this was probably due to the cbmbinatibri of a fast gu'ded 
practice problem followed by tvvo slower prdblertl solving pirbblems. Thus, our subjects did 
not appear to benefit from working through an exampit; before attempting Independent 
solutions. 

CbhclUsibri 

We began this paper by noting that the discovery learning strategy leads to different 
principles for manual design than learning by example. I'he results of this experiment force 
us to reconsider both strategies;. 

With respect to learning by example, the results reported here are incdnsistent with thdsa 
of Sweiler and Cooper (1985), who advocated the study of wdrked-but examfDies over pure 
problem solving for learning a cdgnitive skill. Although the training paradigms thai we used 
differed somewhat from Sweller and Codper's, the cdmparisori Is still apt. We found that 
perfdrmance a; test was consistently better when training consiuted of problem solving ihan 
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when it presented wdrked-dut examples. 
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Even when examples were combined with problem 



sbiving, iDerfbrmarice ai tes* was not significantly better ihan with pure problem so'ving; The 
difference between bur results and Swelier and Cooper's may be explain?d in two ways; 
First; the difference might be attributable to the delay we Imposed between training and the 
test. Swelier and Cooper's subjects were always tested immediately after the training 
session, while bur subjects were tested after a two-day delay. Conceivably, ii is easier and 
more effective to use an example as a model than to generate a solution to a problem, but 
this advantage disappears with time as examples become increasingly difficult to retrieve 
from memory. This woujd sugr^esc that when the test immediately follows training, subjects 
would be able tc retrieve the example and perform better than subjects who only have 
trained with problem solving. However, after a delay, the example may no longer be 
retrlevabre. In this case, all subjects are forced to make use of problem solving techniques 
and sub'ects who have had practice applying the procedures with problern sbiving w! 
perform better. 

The second explanation has to do with the demands of the tasRs. Swelier and Cooper 
were dealing with algebra, where the basic operations were already well-learned by the 
subjects and where distinct problem types may be easily isolated; Studying examples may 
have been more advantageous than problem solving in this situation since the major 
challenge is learning to recognize the problem types. In contrast, learning to select and 
execute computer commands requires learning new operations as well as recognizing the 
situations in which they are appropriate. In this case, studying examples, or ever working 
through step-by-step instructions, is apparently less useful than generating commands 
independently: 

The relatively poor performance in the Guided Practice condition is consistent witi Carroll 
et al.'s (1985) finding that step-by-step iuibrial iristraction is inadequate; Hbwever, our 
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results suggest that the goal-setting and explbratibn aspects of the disccvery learning 
strategy, as ernployed In Carroll et al.'s Guided Expldratidri materials; may not be the 
primary source of the advar<tage they found over training with the commercial tutoria! 
manual A substantial part of the advantage may be due Fiimply to the prdblerh solving 
activity their subjects engaged in; Further research that direcily contrasts problem solving 
and discovery learning will be necessary to determine whether goal setting and exploration 
acuvlties confer additional benefits over and above problem solving. 

We note that our results are limited to prDblems that can be solved by applying a single 
method rather than requiring combinatldns of procedures. It ;s interesting to speculate on 
whether our results will extend to more complex problems. We sometimes noticed that 
while working on training problems in the problem solving condition, subjects would combine 
procedures in inefficient or ineffective ways. After finishing their attempt, they would closely 
consider the solution provided on the .feedback page and draw conclusions about why It 
might be better. We suspect that subjects were less capable of evaluating the solution 
provided in the guided practice condition, since they were not forced to develop an 
independent solutior 'irst. We expect that if subjects are presented with complex problems, 
that allowed several sblutibns of varying efficiency, subjects would again learn "more 
sophisticated sblutibh strategies through problem solving with feedback than through studying 
exarhples or following guided practice problems; 

It is also interesting to speculate on the generality of these findings for different 
pibpulations of subjects. We argued in the introduction that discovery learning may be 
ineffective for peopio who are not familiar enough with typical problem situations in a 
domain to set reasonable goals; What if the subjects were famiiiar with some aspect of the 
domain? Carroll et al;*s (1985) subjects were secretaries learning to use a word processor. 
They were inexperienced cdmpUter Users. bUt highly experienced at the types of tasks they 
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would use the computer for (i:e:; typing letters and manascriptsj; in ihis car;e; the subjects 
may have been able to invent realistic problem situations for themselves that focused their 
exploration on tt, ^ relevant features of the system and helped them invent plausible problems 
to try to solve. In contrast, dUr subjects were learning a new compuier application without 
any real knowledge of the uses to which a spreadsheet might be put. As a result, bUr 
subjects may have been completely lost if left to explore the spreadsheet on their own* the 
training problems clarified the goals to which the procedures might be applied as well as 
the details of actually executing the procedures This may be why we found no interactions 
bstween previous computer experience and training condition in our study: while the 
experienced Users were better at learning id execute the commands 'Efficiently, both groups 
of subjects had to leorri about the •^ituati^ns In which a cdmmand was apprupriate. 

In any case, the results of this sfjdy do not bodn well for interactive tutorial manuals in 
their present fdrm, since most such manuals rely exclusively on guided practice. Novice 
"sers working through such tutdrials may successfully cdmplete al! of the exercises and stllf 
net iearn what they need to use the commands on their own. We believe that adapting the 
Problem Solving form of training developed for this study and distributing practice will 
produce much more effective user manuals; 
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APPENDIX A 



A Typical Manual Entry for a Difficult Gorrirnahd 

MOVE edtUMN 0R ROW 

The Move cdrrimarid moves the entire row or column that contains the current cell to 
another position on the worksheet. 

PROeEDbRES: 

/M [FRGMj . [TO] [f?ETURNj Moves the contents of row or coiainn 

in the [FROM] coordinate to the row 
or column specified in the [TO] 
coordinate. 

The Move command requires the folldwing information: 

• The FROM Cbordiria;es: The cbordinates of a cell In the row or column that 
you wish tp midve. Vlsicaic autcmaiically fills in the cbbrdiriates bf the current 
cell (e.g., D5) as the FROM cobrdinates. If the current cell ij nbt in the row br 
column you wish to move, type [BKSP] to erast >'',erse coordinates and type the 
coordinates of a cell in the row or column you wai t to move. Then type a 
period. Three periods appear on the edit line. Now you can type the "TO" 
coordinates. 

• The TO Coordinates: The coordinates of a celi specifying the destination of the 
move: The TO coordinates must contain either the same column letter or the 
same row number as the FROM coordinates. The VisiCalc program determines 
whether to move a row or a column by comparing FROM and TO coordinates: 
if the column letter In the two coordinates Is the same, then a row Is moved; If 
the row number is the same, then a cdiurnn Is moved. 

The difference between the FROM and TO cbprdinated tells_ VisiCaic where tb 
put the mbved infbrmatjon. If the FROM c_oordinates (e.g., D5) have the same 
column letter as the TO coordinates (e.g., D3), then the contents bf row 5 will 
rribve up to row 3. If the FROM coordinates (e.g., D5) have the same row 
number as the TO coordinates (e.g.. B5), then the contends of column D v;ill 
move left to column B: 



VisiCalc makes room for the row br column >gu move by shifting the other rows and 
columns over. So moving a column cr row to a new location does not "cover up" any 
other entries. 
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A typical Manual Entry for an Easy Gomrriarid 

BhANK eOMMAND 

The BiariR cbmrtiarid irretrievably erases the entry in the current cejl: 
Procedure s: 

/B [RETURN] Erases the laSel or value in the current cell. 

If you typed /B by mistake, you _can abort the corhrhand as long as you have hot yet 
typed [RETURN]. To undo the IB. type any Rev Qxcept [RETURN|, [ROME] or an arrow 
key. 
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. Notes 

^Recall that the Mixed Practice condition did not present a new [<ind cr problam; in this 
condition, subjects learned about a command by working one Guld-d Practice problem 
foliowed by two Problem Solving problems. 

•'The slower times in the Massed-Problem Solving condition appear to be due to one 
especiar difficult command, the "Titles" command. Of the 10 subjects in this condition 
with mean solution times over 100 seconds. 6 were working oh the Titles command (we did 
not counterbalance how often a particular comnic^nd ^Nas assigned to a condition). GruciaHy, 
the Titles command appeared in this condition a disproportionate number of times: nine 
subjects saw \i in the Massed-Problems Solving condition, but only two subjects saw it in 
the Distributed-Problem Solving condition. 

2 — — — __. __ ._. 

The degrees of freedom differ in the two analyses because two control subjects failed 
to solve any test problems correctly. Lata for ihese subjects could be included In the 
accuracy analysis but not the solution time analysis. 
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TABLE 1 



VisiCalc Cbirimands Presented in the Training Manual* 
Classified by Difficulty 



EASY COMMANDS 

/B Blank a ceii 

/D Lelete a row or column 

/P Format a cell's entry 

/T Transfix column/row headers 

/- Fill a cell with characters 

/GC Change width of columns 



DIFFICULT COMMANDS 
(type) Enter value or label in a cell 
/S Edit the entry in a cell 
/M Move a column or row 
/PF Create a printfiie on disk 
/R Replicate a cell or cells 
/W Split window 
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TABLE 2 



Mean Accuracy Scores for Solving Practice Problems 
During Traihingj as a. Function of 
Form of Practice and Spacing. 



FORM OF PRACTICE 



SPACING 



Guided 
Prac t ice 



Mi?:ed Problem-Solving 
Practice Practice 



Massed 



.99 



.84 



Dis tribu ced 



.98 



.83 



.78 
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TABbE 3 



Mean Sblutiori Times . (sec) for Practice Problems 
During Trairiirijj as a. Function of 
Form of Practice and Spacing. 



FORM OF PRACTICE 



SPACING 



Guided 
Prac t ice 



Mixed Problem- Solving 

Practice Practice 



CONTROL 
GROUP^ 



Massed 



52.6 



72.4 



97.7 



55.0 



Distributed 



52.5 



71.2 



88.3 



56.5 



^ The times for the Control Group represent the time a subject spent 
studyingan example problem and its solution without typing at the keyboard 



at any time. 
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TABLE 4 

Mean Accuracy Scores at Test, 
as a Function of Form of Practice and Spacing; 

FORM OF PRACTICE 



Guided Mixed Problem^Solvin^ CONTRCL 

SPACING Practice Practice Practice GRG!5P 

^^assed .47 .65 .64 .49 

Distributed .59 ;67 .;i .45 



B7 



TABLE 5 



Me?.n Sblutiph Times_(sec) at Test, 
as a Fur.ction of Form of Practice and Spacing. 



FORM OF PRACTICE 



SPACING 



Guided 
Practice 



Mixed Prdbler.^-Solvip.g CONTROL 

Practice Practice GROUP 



Massed 



89. 1 



iiO.2 



106.6 



134.5 



Dis triouted 



92.6 



80. i 



62.7 



135.0 



ERIC 



38 



Figure Title 



FIGURE 1: Typical Practice Problem: Presented in (A) Guided Practice For and 
(B) Problem Solving Form, each seen in conjunction wirh VisiCalc Spreadsheet 
on screen as in (C) . 
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A. GciiDED PRACTICE FORM 



Alphabetize the names by putting the rbv^ cbritainirig Sroel.:. and Stevai t 
further down in the appropriate spots: Start with ceii Al as the cirrent 



B. PROBLEM SOLVING FORM 

Alphabetize the names by p-jtting the rows containing Steele and Stewart 
further down in the approoriate spots. 

feedback, appearing on the foJJowing page: 

You could have Used the following sequence of commands (starting with cell 
Al as Mi3 curr-rit cell) to solve the preceding problem. 



cell. 



TYPE TlilS: 



/M . A"7 i Return ] 

/M . A7 [RETURN] 



/M . A7 [RETURN] 
/H . A7 [RETURN] 



C. CONTENTS OF VISICALC DISPLAY IN INITIAL PROBLEM STATE 



A 

1 Steele 



Clerk 
Clark II 
Manager 



B 



2 Stewart 



3 Sanders 

4 Schiff 

5 Sebert 



Accoun tan L 
Sec ' y 
Clerk III 



Manager 



6 Snyder 

7 Sweet 
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